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Executive  Summary 

Initial  guidelines  for  minimum  requirements  for  system  representations  need  to  include 
standardized,  disciplined,  system  engineering  methods  and  processes.  The  Capability  Test 
Methodology  (CTM)^  describes  methods  and  processes  for  development  of  a  Joint  Operational 
Context  for  Test  (JOC-T)  based  upon  authoritative  sources  and  leveraging  Department  of  Defense 
(DOD)  Architecture  Framework  (DODAF)  artifacts.  These  artifacts  evolve  through  a  systems 
engineering  process  that  represents  systems  in  logical  and  physical  designs.  Regardless  of  the 
specific  composition  of  the  design  or  the  collection  of  required  artifacts,  system  engineering 
principles,  methods  and  processes  are  vitally  important.  The  JOC-T  needs  to  have  sufficient  detail 
to  enable  a  comparison  with  the  developed  live,  virtual,  constructive  distributed  environment 
(LVC-DE)  to  satisfy  user  validation.  System  representations  should  be  compatible  with  existing  or 
future  infrastructure  and  should  maintain  a  consistent  interface  throughout  the  development  cycle. 
Candidate  systems  representation  requirements  will  include  adequate  documentation  covering 
interoperability,  compatibility,  security  based  upon  DOD  Information  Assurance  Certification  and 
Accreditation  Process  (DIACAP)  and  Verification,  Validation  and  Accreditation  (VV&A). 

The  JOC-T  is  validated  by  comparison  to  the  referent  authoritative  sources  (e.g.,  Joint 
Capabilities  Integration  and  Development  System  [JCIDS],  Joint  Operations  Concept  [JOpsC] 
family,  Analytic  Agenda,  concept  of  operations  [CONOPS]  documents,  joint  capability  areas 
[JCA],  Universal  Joint  Task  Lists  [UJTL],  etc.).  The  fidelity  of  the  eventual  LVC-DE  should  be 
correlated  to  the  evaluation  strategy  so  as  to  tailor  system  representations  to  meet  test  and  user 
validation  requirements. 

•  Recommended  Guidelines  -  LVC  system  representations  should: 

-  Be  compatible  with  the  current  (or  planned)  network  infrastructure. 

-  Provide  documentation  on  the  following: 


'  The  JME  is  an  instantiation  of  the  Joint  Operational  Context  for  Test  (JOC-T)  using  the  Live,  Virtual,  Constructive 
Distributed  Environment  capability.  A  joint  mission  environment  is  achieved  when  all  required  aspects  of  the 
JOC-T  are  present  or  accurately  represented  (live,  virtual,  or  constructive). 

^  The  CTM  provides  guidance  on  designing  and  executing  system  of  systems  tests  in  the  JME  to  produce  high  quality 
capability  assessments  and  evaluations  supporting  Department  of  Defense  development  and  investment  decisions. 
CTM  can  involve  developmental  or  operational  testing  during  multiple  phases  of  the  acquisition  lifecycle,  including 
concept  refinement,  technology  development,  and  system  development. 
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o  Interoperability  standards,  problems  and  issues;  system  compatibility  with 

infrastructure  requirements  (Distributed  Interactive  Simulation  [DIS],  high  level 
architecture  [HLA],  test  and  training  enabling  architecture  [TENA],  etc.) 

o  WA 

o  Security  based  upon  DIACAP 

Be  able  to  generate  and/or  accommodate  the  required  data  elements  necessary  to  satisfy 
evaluation  measurements  (including  instrumentation  and  tools). 

Be  based  upon  a  JOC-T  derived  from  authoritative  sources  and  using  standardized, 
disciplined  system  engineering  methods,  processes  and  products  (e.g.,  DODAF  artifacts). 

Provide  quantitative  fidelity  descriptions  if  the  representation  must  produce  critical 
parameters  to  specified  levels  of  accuracy  and  precision. 

Use  comparison  to  similar  accredited  simulations  as  a  basis  for  defining  the  fidelity  aspects 
of  representational  requirements. 

Limit  the  fidelity  required  and  implemented  to  what  is  actually  needed  for  user  validation 
and  evaluation  requirements. 
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Recommendations  Regarding  Initial  Guidelines  for  the 
Minimum  Requirements  Necessary  for  System  Representations 
Used  in  the  Joint  Mission  Environment  (JME) 


Testing  in  a  Joint  Environment  Roadmap 

The  Testing  in  a  Joint  Environment  Roadmap,  Final  Report,  Annex  B  discussed  overarching 
requirements  for  system  representations: 

Constructive  and  virtual  representations  of  acquisition  systems  must  be  developed  to 
be  compatible  with  thejoint  mission  inffastmcture  architecture.  . . .  Most  importantly, 
it  should  be  realized  that  a  system’s  interface  should  be  consistent  throughout  all  these 
acquisition  phases,  regardless  of  whether  it  is  a  live,  virtual,  or  constmctive 
representative.  ...  [in  addition]  it  will  be  necessary  to  establish  a  comprehensive,  yet 
efficient,  accreditation  process  to  ensure  that  existing  capabilities  are  adequate  for  joint 
force  assessments.  Accreditation  of  individual  models,  as  well  as  accreditation  of  the 
interactions  required  of  those  models,  will  be  necessary. 

•  Recommended  Guideline:  Constructive  and  virtual  system  representations  should  be 
compatible  with  the  current  (or  planned)  network  infrastmcture. 


DOD  Acquisition  Planning  and  the  CTM 

The  DOD  uses  a  capability-based  planning  (CBP)  process  as  the  principal  decision  support 
process  for  transforming  the  military  forces  to  support  the  national  military  strategy  and  the 
defense  strategy.  JCIDS  is  one  component  of  the  CBP  process.  JCIDS  plays  a  key  role  in 
identifying  the  capabilities  required  by  the  warfighters  to  support  the  National  Defense  Strategy 
and  the  National  Military  Strategy,  but  successful  delivery  of  those  capabilities  relies  on  the  JCIDS 
process  working  in  concert  with  the  other  joint  and  DOD  decision  processes  encapsulated  in  CBP. 
Documentation  developed  (such  as  joint  capabilities  document  [JCD],  initial  capabilities  document 
[ICD],  capability  development  document  [CDD],  capability  production  document  [CPD],  and  joint 
doctrine,  organization,  training,  materiel,  leadership  and  education,  personnel,  and  facilities 
[DOTMLPF]  change  recommendation  [DCR])  during  the  JCIDS  process  provides  the  formal 
communication  of  capability  gaps  between  operators  and  the  acquisition,  test  and  evaluation,  and 
resource  management  communities.  These  documents  provide  the  integrated  architecture  products 
that  ensure  understanding  of  the  linkages  between  capabilities  and  systems  and  aid  acquisition 
decision-making;  and  their  performance  attributes,  including  key  performance  parameters  (KPP) 
and  key  system  attributes  (KSA)  that  define  the  most  critical  elements  of  performance  for  the 
systems  under  development."^  Implementation  of  the  CTM  leverages  these  JCIDS-derived 
products  and  attributes  to  define  the  capability  to  be  evaluated,  and  to  develop  a  realistic  JME  in 
which  to  test.  During  CTM  processes  the  systems,  components,  equipment,  data,  and  other  items 
that  will  compose  the  system  of  systems  (SoS)  are  summarized;  and  the  capability  to  be  evaluated 


^  Director,  Operational  Test  &  Evaluation,  “Testing  in  a  Joint  Environment  Roadmap,  Strategic  Planning  Guidance,  Fiscal 
Years  2006-201 1,  Final  Report,”  November  12, 2004,  B-9. 

^  CJCSI  3170.01F,  Joint  Capabilities  Integration  and  Development  System,  Enclosure  A,  1  May  2007. 
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is  explained.  This  results  in  an  initial  description  of  the  mission  objectives  and  details  the 
high-level  mission  concept. 

Determining  what  systems  are  needed  for  the  joint  mission  or  task  is  the  first  order  of  business 
in  the  CTM  process.  DOD  defines  a  capability  as  the  ability  to  achieve  a  desired  effect  under 
specified  standards  and  conditions  through  combinations  of  ways  and  means  to  perform  a  set  of 
tasks.  It  is  defined  by  an  operational  user  and  expressed  in  broad  operational  terms.  Available 
system  capability  documents  should  provide  the  SoS  description.  The  appropriate  document  used 
for  this  description  will  depend  upon  whether  the  capability  to  be  evaluated  is  a  materiel  or  a 
non-materiel  solution,  and  the  maturity  of  the  system/SoS  development  process.  The  system 
documents  provide  an  overview  of  the  capability  gap  in  terms  of  the  relevant  range  of  military 
operations  and  the  timeframe  under  consideration.  They  also  explain  the  capability  the  system/SoS 
delivers  and  how  it  relates  to  the  key  characteristics  identified  in  the  JOpsC  and  applicable 
CONOPS,  and  provide  integrated  architectures.  The  JCAs,  analytical  baselines,  and  UJTLs 
provide  context  and  references  for  the  system/SoS.  The  capability/SoS  description  summarizes  the 
systems,  components,  equipment,  and  data  that  will  compose  the  SoS,  and  details  how  the  SoS 
contributes  to  the  required  capability,  the  operating  environment  of  the  SoS,  how  the  capability 
will  be  employed  on  the  battlefield,  and  where  it  will  be  employed.  The  initial  system/SoS 
descriptions  provide  the  key  features  and  subsystems,  hardware,  and  software  for  each  increment’s 
configuration.  By  definition,  a  “system  of  systems  (SoS)’’  is  “A  set  or  arrangement  of 
interdependent  systems  that  are  related  or  connected  to  provide  a  given  capability.  The  loss  of  any 
part  of  the  system  will  significantly  degrade  the  performance  or  capabilities  of  the  whole.’’^  Thus 
to  fully  describe  the  capability,  the  SoS  must  be  decomposed  into  testable  systems  that  can  be 
integrated  into  the  JME. 

In  addition  to  the  System  Engineering  processes,  the  CTM  develops  an  Evaluation  Strategy 
based  upon  three  levels  of  measurements:  1)  measure  of  system/SoS  attributes  (MOSA);  2)  task 
measures  of  performance  (TMOP);  and  3)  mission  measures  of  effectiveness  (MMOE).  This 
measures  framework  supports  resolution  of  critical  operational  issues  (COIs).  This  measures 
framework  produces  a  number  of  products  including  an  Integrated  Data  Requirements  List 
(IDRL).  System  representations  need  to  be  able  to  generate  and/or  accommodate  the  required  data 
elements  to  satisfy  the  IDRL. 

•  Recommended  Guideline:  Provide  documentation  on  the  following: 

-  Interoperability  standards,  problems  and  issues;  system  compatibility  with  infrastructure 
requirements  (DIS,  HLA,  TENA,  etc.) 

-  VV&A 

-  Security  based  upon  DIACAP 

•  Recommended  Guideline:  System  representations  need  to  be  able  to  generate  and/or 

accommodate  the  required  data  elements  necessary  to  satisfy  evaluation  measurements 

(including  instrumentation  and  tools). 


^  CJCSM  3170.01C,  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,  Glossary,  Part  II  - 
Definitions,  I  May  2007. 
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The  JOC-T  Development  Process 

The  JOC-T  describes  the  overall  philosophy  of  forces  operating  jointly  and  the  tactics, 
techniques,  and  procedures  (TTP)  to  be  employed  to  achieve  effects  on  the  battlefield  by  exhibiting 
capabilities  they  will  not  possess  separately.  Developing  the  JOC-T  includes  performing 
operational  analyses  to  constrain  the  test  into  a  manageable  set  of  systems  and  operational 
interactions.  It  is  derived  from  the  JME  in  which  the  system/SoS  is  designed  to  operate  and 
provides  the  mission  statement,  end  states,  mission  objectives,  blue  forces,  threat,  and 
environmental  aspects  which  provide  a  foundation  for  the  test  scope.  This  composition  allows  for 
VV&A  of  the  JOC-T  by  checking  for  consistency  across  interaction  descriptions,  and  identifying 
operational  context  gaps  through  traceability  analyses  back  to  authoritative  sources  and  test 
customer  capability  evaluation  needs.  The  focus  on  JOC-T  ensures  the  testing  will  focus  on  the 
overall  structure,  major  elements,  and  objectives  of  the  test  and  evaluation  (T&E)  program  with 
relation  to  the  operational  context.  The  JOC-T  can  then  be  used  in  follow-on  development  of  the 
test  scenario,  vignettes,  and  trials.  The  processes  involved  in  creating  the  JOC-T  are  shown  in 
figure  1. 


DOD  Architecture  Framework  Artifacts  in  the  JOC-T 

The  JOC-T  document 
includes  several  operational 
and  system  views.  The  OV-1 
is  a  high  level  mission  graphic 
that  portrays  key  joint 
capability  effects  to  include 
the  mission  statement,  end 
state,  the  key  blue  and  threat 
forces  as  well  as  specified 
tasks.  The  blue  operational 
activity  model  (blue 
operational  view-5,  [BOV-5]) 
describes  the  activities 
through  which  each  military 
mission  is  accomplished.  The 
blue  operational  node 
connectivity  description 

(BOV-2)  identifies  the 
operational  nodes  and 

information  flow  lines  between  the  blue  forces.  The  blue  organizational  relationships  chart 
(BOV-4)  should  also  be  included  in  order  to  show  the  task  organization  of  the  blue  forces.  This  is 
vital  to  understanding  the  relationships  among  the  organizations  which  comprise  the  blue  force. 
The  blue  SoS  interface  description  is  captured  in  the  blue  systems  view  (BSV-1)  which  graphically 
illustrates  data  elements  and  data  characteristics  and  the  relationships  among  them.  The  blue 
operational  activity  to  systems  functionality  traceability  matrix  (BSV-5)  shows  the  SoS  operational 
activity  to  SoS  function  traceability  matrix;  this  depicts  the  mapping  between  the  capabilities  and 
systems  and  identifies  the  transition  of  a  capability  into  a  planned  or  fielded  system.  The  systems 


Figure  1.  Joint  Operational  Context  for  Test  Development 
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functionality  description  (BSV-4)  describes  the  system  funetions  as  well  as  the  data  produeed  and 
consumed;  it  deseribes  the  how,  what,  and  the  relationships  between  the  blue  forces.  The  final 
systems  view  is  the  BSV-6,  which  contains  the  details  on  the  arehitecture’s  systems  data  elements 
and  their  attributes  grouped  under  the  title  Data  Exchange.  The  systems  data  exehange  matrix 
provides  the  systems  details  that  implement  the  operational  information  exchange  requirements 
and  relates  these  data  elements  to  systems  nodes,  system  funetions  (eontained  in  the  SV-4)  and 
time.  In  most  eases  the  JCIDS  arehiteetural  produet  direetly  translates  into  the  CTM  blue 
arehiteetural  produet.  At  other  times  (depending  upon  the  system  development  maturity),  some 
data/information  may  need  to  be  colleeted  from  the  applieable  JCIDS  doeuments  and  the  DODAF 
products  updated  or  modified  to  produce  the  eorresponding  blue  product. 


Environmental  Factors 

By  completing  the  operational  and  systems  views  the  tester  is  able  to  determine  the  blue 
aetions,  the  sequence  diagrams  and  phasing  of  maneuver,  the  interactions,  and  the  environment 
effeets.  In  order  to  fiilly  understand  the  SoS,  it  is  necessary  to  deseribe  the  context  in  whieh  the 
SoS  will  be  employed.  The  operational  environment  addresses  all  operational  requirements  and 
specifieations  required  of  the  SoS  to  inelude  the  individual  system  platforms.  It  establishes  the 
geographieal  and  environmental  eonditions  in  whieh  the  SoS  will  operate.  Different  operating 
environments  and  their  eharacteristies  will  affeet  support  requirements. 

When  describing  the  environment  in  which  the  SoS  will  operate,  it  is  important  to  discuss  both 
the  physical  and  civil  environmental  aspeets.  The  environment  ineludes  the  air,  water,  land,  plants, 
animals,  and  other  living  organisms,  manmade  struetures,  historical  and  cultural  resources,  and  the 
interrelationships  that  exist  among  them.^  Analysis  of  the  environment  involves  examining 
environmental  eonditions  that  are  potential  test  and  evaluation  faetors,  and  should  be  identified  as 
input  into  the  development  of  the  evaluation  strategy.  Environmental  conditions  are  broken  down 
as  either  physical  or  civil. 

The  physieal  environment  includes  both  natural  and  man-made  environments,  and  eould 
extend  to  geospatial,  meteorologieal,  oeeanographic,  and  space.  Physical  environment  ean  include 
both  external  and  internal  conditions  (such  as  temperature,  humidity,  radiation,  magnetie  and 
electrie  fields,  and  shoek  vibration).  SoS  environment  descriptions  should  include  SoS  infiuenees 
that  would  affect  the  performance,  reliability,  or  survivability  of  the  SoS.  If  key  to  the  testing  of 
the  SoS,  the  physieal  environment  description  should  include  possible  modes  of  transportation  into 
and  within  expected  areas  of  operation  and  existing  infrastructure  support  capabilities. 

Civil  eonditions  include  local  indigenous  eustoms,  eeonomie,  ethnic,  political  and  religious 
factions  or  groups,  group  history  and  inter-relationships,  general  population  views  toward  blue  and 
threat  forees,  ete. 


®  CJCSM  3170.01C,  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,  Glossary,  Part  II  - 
Definitions,  I  May  2007. 
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Threat  Representations 

The  final  portion  of  the  JOC-T  is  the  threat  foree  deseription.  Specifically,  it  contains  the 
recommended  threat  operational  views  (TOV)  to  be  created.  The  system/SoS  threat  assessment 
should  include  threat  force  descriptions.  These  descriptive  products  can  include  a  threat  order  of 
battle,  threat  system/SoS  descriptions  (functional  and  physical  threat  system  views),  threat 
command  and  control  (C2)  structure  operational  node  connectivity  description  view  (TOV-2), 
threat  actions  operational  activity  model  (TOV-5),  threat  maneuver  scheme/phasing,  threat  force 
lay  downs,  and  threat  interactions.  The  TOV-1,  TOV-4,  and  TOV-5  are  used  to  show  the 
organizational  relationship,  the  C2  structure,  and  the  SoS  interface  and  functionality.  The 
description  of  the  threat  forces  should  include  the  interactions  of  the  threat  with  blue  forces,  with 
other  threat  forces,  and  with  the  environment. 

Threat  actions  include  joint/Service  task  decompositions,  mission  threads,  and  threat 
operational  activity  flows.  Descriptions  include  key  blue  to  threat,  threat  to  threat,  and  threat  to 
environment  interactions  with  potential  testing  implications.  The  threat  high-level  graphic  should 
focus  on  the  threat  aspects  for  the  SoS.  The  Threat  Force  OV-1  describes  a  mission  and  the  main 
operational  nodes,  ft  should  depict  any  unique  aspects  of  threat  force  operations  as  well  as  the 
interactions  between  threat  forces,  their  architecture,  and  the  environment.  The  purpose  of  the 
OV-1  is  to  give  a  high-level  description  of  what  the  threat  forces  are  supposed  to  do  and  explain 
how  this  will  be  achieved.  The  textual  portion  of  the  OV-1  should  explain  the  top-level 
operational  role  of  the  threat  forces  shown  in  the  graphic  and  the  context  within  which  they  are 
performing  their  mission. 

•  Recommended  Guideline:  System  representation  requirements  should  be  based  upon  a 
JOC-T  derived  from  authoritative  sources  and  using  standardized,  disciplined  system 
engineering  methods,  processes  and  products  (e.g.,  DODAF  artifacts). 

WA  oftheLVC-DE 

The  validation  of  the  JOC-T  uses  the  aforementioned  authoritative  sources  as  referent.^  The 
LVC  is  then  validated  against  the  JOC-T  as  referent.  The  key  notion  here  is  the  “tesf  ’  part  of  the 
JOC-T;  this  means  the  JOC-T  represents  a  specific  scope,  resolution,  and  context  for  LVC 
development  needed  for  test. 

Scope  is  concerned  with  the  range  of  parameters  or  applications  of  concern  for  the 
referent. 

Resolution  is  concerned  with  the  level  at  which  distinctions  can  be  made  in  the 
information  of  the  referent.  . .  .Resolution  of  information  in  the  referent  must  be  at 
the  lowest  level  required  to  support  M&S  validation  assessment. 

Context  addresses  the  environment  within  which  the  referent  information  is 
applicable.  Sometimes  the  context  is  simply  a  set  of  assumptions;  sometimes  the 
context  is  a  physical  condition  (such  as  pressure).  There  is  a  difference  between 

’  “The  referent  is  the  best  or  most  appropriate  codified  body  of  information  available  that  describes  characteristics  and 
behavior  of  the  reality  represented  in  the  simulation  from  the  perspective  of  validation  assessment  for  intended  use  of  the 
simulation.”  Pace,  D.K.  “The  Referent  Study  Final  Report,”  Defense  Modeling  &  Simulation  Organization,  June  2004, 

p.  6. 


7 


context  and  scope.  Context  is  not  specified  in  the  variables  used  to  represent 
entities,  processes,  and  interactions  described  by  the  information  in  the  referent,  but 
is  concerned  with  parameters  and  factors  that  might  influence  the  measurements 
(values)  of  the  referent  information.* 

Using  the  JOC-T  as  referent,  the  LVC  seeks  to  achieve  a  level  of  fidelity  that  will  meet  the 
user  requirements  for  validation.  In  order  to  meet  these  requirements,  the  “. . .  referent  must  be 
carefully  defined  in  terms  of  how  much  is  to  be  simulated  (i.e.,  entities  and  characteristics)  and 
what  interactions  are  involved  (i.e.,  relationships  between  entities  in  the  referent).”^  The  JOC-T 
development  process  is  intended  to  satisfy  this  fidelity  measurement  requirement. 

Required  fidelity  will  vary  depending  on  the  evaluation  focus.  If  the  representation  must 
produce  critical  parameters  to  specified  levels  of  accuracy  and  precision,  then  system 
representations  should  provide  quantitative  fidelity  descriptions.'^  Figure  2  shows  a  Ifamework 
for  understanding  and  applying  fidelity;  it  “. . .  clarifies  the  difference  between  the  fidelity  required 
by  the  application  (captured  in  the  simulation  requirement),  and  the  fidelity  present  in  a  specific 
model  or  simulation  (contained  with  the  M&S  capabilities).  Both  the  fidelity  required  and  the 
fidelity  present  is  characterized  in  terms  of  resolution,  error/accuracy,  sensitivity,  precision  and 
capacity.”" 


Figure  2.  Framework  for  Understanding  and  Appiying  Fideiity 

Fidelity  descriptions  that  use  quantitative  data  fall  into  the  category  of  “long  descriptions.” 

Long  descriptions  of  simulation  fidelity  typically  describe  simulation  fidelity  in 
terms  of  multiple  explicit  attributes.  The  number  and  kinds  of  attributes  considered 


*  Ibid.,  p.  14. 

®  Defense  Modeling  &  Simulation  Office,  “Recommended  Practices  Guide  Special  Topic:  Fidelity,”  2000.  p.  6. 

Ibid.,  p.  5. 

"  Ibid.,p.  7-9. 
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varies  with  the  construct  being  employed  for  simulation  fidelity.  Most  constructs 
consider  either  the  scope  of  the  simulation’s  treatment  of  significant  factors  in  the 
application  domain  (this  usually  involves  some  kind  of  enumeration),  the  quality  of 
treatment  of  factors  within  the  simulation  (as  indicated  by  parameter  accuracy, 
resolution,  etc.),  or  both. 

In  figure  2,  the  M&S  fidelity  description  is  compared  to  the  application  tolerances  (coming 
from  the  JOC-T  and  the  evaluation  strategy)  to  render  a  “fitness”  of  the  representation  in  the  JME. 
The  evaluation  strategy  is  key  to  whether  or  not  a  representation  is  live,  virtual  or  constructive.  The 
resolution,  error/accuracy,  sensitivity,  precision  and  capacity  that  are  required  to  satisfy  the 
measurements  in  the  evaluation  strategy  will  dictate  the  choices  of  fidelity  made  in  LVC 
development.  Further  decisions  based  upon  cost  and  schedule  will  translate  into  the  degree  of  risk 
accepted  by  the  program  manager/tester.  It  is  useful  to  compare  the  representation  to  simulations 
meeting  similar  purposes  in  order  to  gauge  its  fitness  for  purpose. 

•  Recommended  Guideline:  If  the  representation  must  produce  critical  parameters  to  specified 
levels  of  accuracy  and  precision,  then  system  representations  should  provide  quantitative 
fidelity  descriptions. 

•  Recommended  Guideline:  Use  comparison  to  similar  accredited  simulations  as  a  basis  for 
defining  the  fidelity  aspects  of  representational  requirements. 

•  Recommended  Guideline:  System  representations  should  limit  the  fidelity  required  and 
implemented  to  that  which  is  actually  needed  for  user  validation  and  evaluation 
requirements. 


Ibid.,  p.  4. 
‘^Ibid.,p.  10. 
Ibid. 

Ibid. 
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Recommended  Guidelines 


Live,  virtual,  constructive  (LVC)  system  representations  should: 

1 .  Be  compatible  with  the  current  (or  planned)  network  infrastructure 

2.  Be  based  upon  a  JOC-T  derived  from  authoritative  sources  and  using 
standardized,  disciplined,  system  engineering  methods 

3.  Provide  documentation  on  Interoperability,  VV&A  and  DIACAP 

4.  Be  able  to  generate  and/or  accommodate  required  data  elements 
necessary  to  satisfy  evaluation  measurements 

5.  Provide  quantitative  fidelity  descriptions  if  the  representation  must 
produce  critical  parameters  to  specified  levels  of  accuracy 

6.  Use  comparison  to  similar  accredited  simulations  as  a  basis  for 
defining  the  fidelity  aspects  of  representational  requirements 

7.  Limit  the  fidelity  required  and  implemented  to  that  which  is  actually 
needed  for  user  validation  and  evaluation  requirements 

DIACAP  -  DOD  Information  Assurance  Certification  and  Accreditation  Process  JOC-T  -  Joint  Operational  Context  for  Test 
W&A  -  Verification,  Validation,  and  Accreditation 
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Testing  In  a  Joint 
Environment  Roadmap  (TIJR) 


1 .  Be  compatible  with  the  current  (or  planned) 
network  infrastructure 

-  TIJR  discussed  overarching  requirements  for 
system  representations: 

“Constructive  and  virtual  representations  of  acquisition 
systems  must  be  developed  to  be  compatible  with  the  joint 
mission  infrastructure  architecture... .Most  importantly,  it 
should  be  realized  that  a  system’s  interface  should  be 
consistent  throughout  all  these  acquisition  phases,  regardless 
of  whether  it  is  a  live,  virtual,  or  constructive  representative...” 


3 


Aoon^ 


Joint  Operational  Context 
for  Test  (JOC-T) 


2.  Be  based  upon  a  JOC-T  derived  from 
authoritative  sources  and  using  standardized, 
disciplined,  system 

-  Blue  Forces 

-  Threat  Forces 

-  Environment 
(Physical  &  Civil) 

-  Conceptual  Model 

-  Logical  Design 

-  Physical  Design 

-  DODAF  Artifacts 


engineering  methods 


Analytic  Agenda 
(OPS,  MSFO,  Analytical  Baseline) 


COCOM  IPL 


Task  Lists 
(UJTL,  JMETL.  Service) 


STAR 


Key  DOD 

Capability  Based  Planning 
Inputs  to  JOC-T 


JOpsC  Family 
(cao,  Joc,JFC,  Jiq 


JCIDS 

(JCD,  ICO,  COD,  CPD,  OCR) 


Systems  Engineering 
Documents 


CCJO  -  Capstone  Concept  for  Joint  Operations  CDD  -  Capability  Development  Document  COCOM  -  Combatant  Command  CPD  -  Capability  Production  Document 

DCR  -  DOTMLPF  Change  Recommendation  DODAF  -  DOD  Architecture  Framework  DPS  -  Defense  Planning  Scenario  ICD  -  Initial  Capability  Document  IPL  -  Integrated  Priority  List 

JCA  -  Joint  Capability  Area  JCD  -  Joint  Capabilities  Document  JCIDS  -  Joint  Capabilities  Integration  &  Development  System  JFC  -  Joint  Force  Commander  JIC  -  Joint  Integrating  Concept 

JMETL  -  Joint  Mission-Essential  Task  List  JOC  -  Joint  Operating  Concept  JOpsC  -  Joint  Operations  Concepts  MSFD  -  Multi  Service  Force  Deployment  STAR  -  System  Threat  Assessment  Report 

UJTL  -  Universal  Joint  Task  List 
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Documentation 


3.  Provide  documentation  on  Interoperability, 
VV&A,  and  DIACAP 

-  Interoperability  standards, 
problems  and  issues;  compatibility 
with  infrastructure  requirements 
(DIS,  HLA,  TENA,  etc.) 

-  System  W&A  reduces  the  risk 
to  overall  LVC-DE  validation 

-  DIACAP  requirements  can  be  a  “long  pole”  in  the  tent 
(JBD2  lesson  learned) 

Important  inputs  for  successful  instantiation  of  the  LVC-DE 

DIACAP  -  DOD  Information  Assurance  Certification  and  Accreditation  Process  DIS  -  Distributed  Interactive  Simulation  HLA  -  High  Level  Architecture 
JBD2  -  Joint  Battlespace  Dynamic  Deconfliction  TENA  -  Test  &  Training  Enabling  Architecture  W&A  -  Verification,  Validation,  &  Accreditation 
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Data  Collection 


4.  Be  able  to  generate  and/or  accommodate 
required  data  elements  necessary  to  satisfy 
evaluation  measurements 


JBD2  test  layers 

Infrastructure: 

Network,  Middleware 


Real  Time/Post-Test 
Processing 


Non 
Real-Time 
Data 
Transfer 


LVC  systems/systems  of  systems 
representing  warfighting  capabilities 


Resource 
Data  Loggers 


Operational  Capability 
Data  Loggers 


"It's  all 
about  the 
data!" 
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Fidelity  Description 


5.  Provide  quantitative  fidelity  descriptions  if  the 
representation  must  produce  critical  parameters 
to  specified  levels  of  accuracy 

-  Qualitative  (High/Medium/Low)  descriptions  lack 
the  information  content  necessary  to  support 
technical  decisions  about  simulation  fitness  for  a 
particular  purpose 

-  Fidelity  characterized  in  terms  of  resolution, 
error/accuracy,  sensitivity,  precision  and  capacity 
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Comparison 


6.  Use  comparison  to  similar  accredited 
simulations  as  a  basis  for  defining  the  fidelity 
aspects  of  representational  requirements 

-  In  the  absence  of  quantitative  methods,  the  fidelity 
of  the  proposed  simulation  can  be  compared  to 
simulations  meeting  similar  purposes  in  order  to 
gauge  its  fitness  for  purpose 

-  Leverage  systems/simulations  that  have  completed 
verification,  validation,  and  accreditation  (VV&A)  if 
the  similarity  is  sufficient 
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Fidelity  Threshold 


7.  Limit  the  fidelity  required  and  implemented  to 
that  which  is  actually  needed  for  user  validation 
and  evaluation  requirements 

-  Using  the  Joint  Operational  Context  for  Test  (JOC-T), 
the  LVC  seeks  to  achieve  a  level  of  fidelity  required 
by  the  evaluation  focus  that  will  meet  the  user 
requirements  for  validation 

-  Higher  fidelity  simulations  cost  more  time  and  money 
to  build,  more  verification  and  validation  (V&V),  and 
more  to  operate 

-  Real  value  of  system  representations  comes  from 
abstracting  away  irrelevant  details 


Avoid  “Gold-Plating” 
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Contacts 


Max  Lorenzo 

JTEM  Joint  Test  Director 
maxinno.lorenzo@ite.osd.nnil 

757.638.6079 

Daniel  Grenier 

JTEM  System  Engineer 
daniel.grenier@ite.osd.mil 

757.638.6021 

Richard  Felsinger 

JTEM  System  Engineer 
rfelsing@scrires.com 

843.218.6217 

itenn@ite.osd.nnil 


Serving  the  testing,  acquisition,  and  warfighting  communities 
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Back-up 


Framework  for  understanding  and  applying  fidelity 
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